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A3STRACT 



Supplemental Course Instruction (SCI) at Olivet 



Nazarene University is describe as a prog ran: m which a high 
achieving student retakes a course successfully completed m order to 
model desired behaviors for other students. The SCI Leader 
demonstrates good study skills and organizes small group study 
sessions. This study examines the effectiveness of the SCI program. 
Confounding factors such as the voluntary nature of the study 
sessions and the open admission policy of the college were controlled 
through structural equation modeling. Structural equation modeling 
requires that all of the hypothesized relationships between the 
variables be posited in advance. A total of * 61 students enrolled in 
four freshman-level courses were surveyed, wi; h complete data being 
obtained for 253 cases which were used in the structural equation 
analysis. The analysis studied: (1) the effect of factors affecting 
SCI participation, such as high school rank, marital status, semester 
load, and expected grade; and (2) the effects of SCI participation on 
course grade, semeste- grade point average, and re-enrollment. The 
study found that the more a student is "at risk" the more likely he 
or she is to use SCI- The study also found a positive impact of SCI 
participation on course grade. There were direct effects of SCI cn 
grade point average, suggesting the transfer of study skills learned. 
(10 references) (JDD) 



* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 



ERJ.C 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



U). 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) " 



SEPARATING TOE WHEAT PROH THE CHAPP: 
FINDING THE UNIQUE EPFFCT OF 
SUPPLEMENTAL COURSE INSTRUCTION 



Jeralynne Hawthorne, John W. Hawthorne 
Olivet Nazarene University 



Prssented at the 11th A 
National Association for 
New Orleans, 



nnual Conference of the 

Developmental Education 
March 6, 1987 



Supplemental Course Instruction at Olivet Naiarene University 



Supplemental Course Instructicn io a program adapted from 
Ihe SuDplemental Instruction program at University of Missouri 
at Kansas City. SCI is an attempt to instill in students the 
appronriate behaviors for success in a particular course by 
modeling those behaviors. Information about the program at UMKC 
can be found in various sources which are cited in Martin et. 



al. (1983). 

In SCI, an undergraduate student is selected from among 
students who have made an A in a given course and have shown an 
ability to communicate with their peers. This oerson, called an 
SCI leader, models desired behaviors while retaking the course. 
The SCI leader demonstrates good study skills in the classroom 
by taking qood notes, not sleepinq, asking the right kinds of 
questions, etc.. 

The leader also organizes study sessions. Study sessions are 
held twice a week for about an hour at a time. Attendance is 
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voluntary. During the study session, the leader directs a review 
of course material by reviewing notes, discussing textbook 
material , trying to help the students discover for themselves 
what may or may not be on a test, directing the groun in writinq 
a possible t2St, and using other technigues described in the 
materials ttom the Supplemental Instruction program at UMKC . 

Our purpose in having SCI is to provide as much assistance 
to students as possible, in a cost efficient way, and to help 
students identified upon entry as marginal or perhaos high risk. 
We also hope to help low risk freshmen to increase the 
likelihood of a successful first year and thereby to increase 
the possibility of their retention. We believe that if a student 
can find success in one class and learn better study skills in 
that class then the skills learned through SCI will transfer to 
other courses . 

The reason for writing this paper is to examine th* 
effectiveness of SCI in liqht of its stated puroose. Aft<>r 
controlling for the confounding effects of marita'i status, ago, 
high school rank, and ACT scores, what is the actual effect of 
attendinq the SCI study sesP.ons on course grade, semester grade 
point 8-eraqe and reenro) iment? All these variables are inter- 
related and it is difficult to sav what the effect of earn 
indi' idual variable may be. A secondary focus of this oao^r is 
on the use of structural equation modeling as a means of describ- 
ing tbe relationship of the variables. 



Evaluating Program Effectiveness 



One concern in evaluating a program is to show what the 
institution would be like if the program did not exist. 
Hopefully, one proves that the institution is different due to 
the program and that the difference is for the better . This 
oositive effect will supposedly lead to continued and, Perhaps, 
increased funding. 

A more idealistic concern in evaluation is to provide 
direction for the program. Does there need to be modification? 
Should there be an expansion of services or maybe a reduction? 

Our intent is to show that SCI is worth the time, money, and 
effort. We want to show that there is a direct impact on the 
student who attends SCI and that that impact is seen in better 
qrades and continued enrollment. 

Most program evaluations are done using some variation of an 
experimental model. As William Cooley (1978) points out, this is 
particularly true within education. Using specified outcome 
measures, a Population that has participated in the program is 
compared with a population that has not. It is then assumed that 
any differences between the "exptf imental " and "control" groups 
are due to the effect of the proaram. The outcomes measures are 
then submitted to certain statistical tests (t-test, analysis of 
variance) to see if the differences that are observed are 
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The true experimental model Is a very powerful explanatory 
cool.. This is true because it has the power to control all th^ 
variables affecting the experimental and control groups exceot 
the treatment effect. Because these two groups are equivalent J 
(either due to matching or random assignment) all differences 
must be attributable to the treatment effect. However, very 
rarely in the social sciences do we have the luxury of 
equivalent grouos . Onlv carefullv designed psych labs 
approximate this. In education r we quickly move away from the 
experimental ideal. Not onlv are we working with qrouDS that are 
not quite equivalent, but the groups vary in a number of wavs 
that can have a direct bearing on the effectiveness of the 
program. Because of these and other problems, Cooley calls for 
us to see 

educational treatments as multl -d imens iona 1 domains, 
and not as distinct, discrete, homogeneous treatments 
worthy of comparison as levels in a typical analvsis 
of covariance contrast" (1978, P. xxiv). 

These considerations must be kept in mind whenever a orogram 

like SCI is evaluated. 

There are a number of factors that affect the success of **tfwii%$&>Vy/ , ^ ^'^r» 

SCI. First of all, students are not required to attend the studv 

sessions. This means that only students who are motivated to put 

out the extra effort will regularly attend sessions. Because 

these students mav have done well in the course anywav, they may 

receive little direct benefit from the program. Another 
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confounding factor is that these sessions are usually held in 
the late afternoon or in the evening. This fact may make it 
difficult for nonreside.it stuoents to oarticipate in the 
Program. A third confounding factor is an ooen admission policy. 
Because we admit students who might be considered "marginal" or 
high risk, it makes it difficult to tell whether to what degree 
a program benefits these students. If we had a perfect 
experimental model r all of the relevant factors would be 
controlled. Nothing would vary exceDt oarticipation in SCI. 
This, however, cannot be done, what can be done, though, is to 
cor.f-rol these factors statistically. This is the singular 
strength of structural equation model ing . 



The Loqic of Structural Equation Modelinq 

In fie S3me way that the classical experimental model is 

really an exercise in design loqic, structural equation modelinq 

is first of all a logical statement. As Anderson and Evans 

(1974) define it, structural equation modeling begins 

with a statement of a verbal theory that makes 
explicit the relationships that are hypothesized amonq 
a set of variables as we 11 as the causal sequence 
thought to exist among them (p. 30). 

It is only after the logical elements of the mcut'i nave been 
assembled and defended that tne statistical work can *->e done to 
"test" the model . 
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This logical model of the phenomena belnq observed beqins 
with the specification of what variables affect the dependent 
variable. These may be variables that are comoletely inside the 
model or those that are outside the model. The key differences 
between these types of variables is time order or sequencing.. 
Pactors which exist prior to the model (age, social class) are 
exogenous. No attempt is made to explain why they mav vary. Thev 
are simplv accepted as background variables. Factors which occur 
slightly before the dependent variable are endogenous We will 
attempt to explain their variance as a result of other variables 
in the model. 

In a Properly designed structural equation model , all of 
th_2 hypothesi zed rela t ionsh ips bet* een the variables are po s i t ed 
in advance. This is why it is a logical exercise. As Blalock 

( 1964 ) puts it, "the basic dilemma is that of how much to 

oversimplify reality". What is actually tested statistically is 
the validity of the hypothetical model created. There are an 
infinite number of models Possible. The selection of one model 
over another is due to the plausibility of the relationships 
within it. As a result, the hypothetical model must be created 
with great care. 

If the model is specified oroperly, and certain a priori 
assumptions hold, the model can be tested with the nse of least 
squares regression. Estimates o f the effects of exogenous ar.d 
endogenous va r i ables on the dependent variable can be computed 
(as well as the : fects of those variables on other endogenous 
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variables). The maqnitude and direction of these estimates can 
then be compared with the oriqinal model to determine its 
validity. 

This approach allows the researcher to determine the unique 
direct and indirect effects of all of the variables in the 
model. If the hypothetical model is not supported, it can be 
revised and the process begun again. Results obtained can then 
be related to certain policy decisions. Certain assumptions 
regarding the program in question may have to be withdrawn or 
reevaluated. New factors, previously thought to be insignifi- 
cant, may deserve attention. 

The logic of structural equation modeling makes it possible 
to estimate the factors that are thought to cause a certain 
program to succeed and thsn test to see if those factors reallv 
are important. However, the largest task is to create the model 
in the first place. 

A Tentative Model of the Effects of SCI Participation 

If the SCI Program accomplishes its objectives, participat- 
ing students should show an increase in course grade, semester 
grade point average, and reenrol lment . The models being tested 
(see Figures 1 and 2) put SCI participation at a pivotal point 
in the causal order between individual background character is- 
tics and these outcome measures. It must be remembered that this 
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is only one of an infinite number of models that could o*» 
t**ted., The rationale for the focus on SCI i« that wo aro 
interested in program evaluation. As a result, we need to be 
able to highlight the Impacts of the program. The more specific j 
rationales for the /oothesized relationships follow. The 
dependent variables will be discussed first, followed by the 
exogenous variables and then the endoaenous variables. 

Dependent Variables . Technically, only two of the outcome 
measures, semester GPA and reenrollment , are treated as depend- 
ent variables. Course grade will be discussed later as an 
endogenous variable which affects both of these dependent 
measures . 

Exogenous Variables . There are a set of factors, occurrinq 
prior to college enrollment, which should significantly affect 
college performance (see Grant and hoeber , 1978). These factors 
can be further divided into two groups, one group dealing with 
academic prowess and the other dealinq with individual 
background characteristics. The first group of variables 
includes high school performance (measured bv reported <*rades 
and class rank) and a risk factor computed from ACT scores. Th*» 

second group includes variables of age (measured as number of A^^SSSi^S^*..^**^-^** 
years since hiqh school) and marital status. 

The endogenous variables will be affected bv th** S o 
"earlier" factors. The better the student was in hiqh school, 
the more likely he will be to expect success in collie 
coursework. This mav make it more likelv for the student to 
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enroll for a heavier load. The student will also be more likely 
to predict high achievement in any particular class. In 
addition, the success in high school will be related to the 
availability of scholarship money and therefore the students 
level of employment. These low risk students will be less likely 
to see extra programs (like SCI) as beneficial but will still 
Derform at a higher level than students who were less oreoared. 
As a result, the models hypothesiie that the low risk, success- 
ful high school students will, a) take heavier loads, be less 
likely to work, c) expect better grades, d) be less likely to 
part iciDate in SCI, e) achieve (both in course grade and 
semester GPA) at a higher level, and f) be more likelv to enroll 
in school in subsequent semesters than higher risk, less 
successful students. 

Students who are older or are married may feel less 
confident about the colleqe settinq. This may cause them to take 
longer to comDlete their education, takinq fewer hours oer 
semester. In addition, the older the student, the more likely 
she would be to work. These added time oressures r«ay cause lower 
expectations of performance and make participation in prog raits 
like SCI difficult. However, these older students nav have an 
added incentive to achieve that often motivates them even if 
they do not match the imaqe of the traditional student. This 
motivation can result in higher performance levels and cont inua- 
tlon in the colleqe proqram. Compared to the trad it ional 
resident student, we would exoect older or married students to 
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a) take lighter loads, b) be more likely to work, c) have lower 
initial grade expectations, d) be less likelv to participate in 
SCI, e) achieve (both in course grade and semester GPA) at a 
higher level, and f) be more likely to enroll in school in 
subsequent semesters. 

Endogenous Variables . Relatic nships can al so oe 
hypothesized aniong the endogenous variables. There are five 
endogenous variables: semester load, hours worked, expected 
course grade, SCI participation, and achieved course grade. Not 
only do each of these relate to the dependent variables, but 
some relate to other endogenous variables as well. 

The more hours a student takes durinq a semester, the less 
likelv he is to be employed. In addition, an increase in course 
load may result in time management problems, forcing the e tud2nt 
to "aim" for a lower grade in the course and to declare 
extrs-course programs as unaffordable luxuries. Having a heavier 
course load could result in lower course grades and therefore 
lower semester CPAs. However, because additional hours taken 
reflect an increased commitment to the institution, there should 
be a positive effect on reenrol lment . 

The same logic would applv with respect to the effect of 
working. The more a student works, the more time pressu? es would 
cause conflicts. This would result in the same predictions for 
expected qrade, SCI participation co e grade, and GPA. One 
difference from that described above is that the student's 
emdoyment is a result of financial pressures, working a larqe 
number of hours rray be related to lower rates of reenro 1 lment . 
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The more a student expects to succeed In a course, the more 
likely she is to do so. There should be a direct relationship 
between expected grade and achieved grade. There should also be 
a direct relationship between expected grade ar.a both of the 
deoendent variables. The exception to this positive pattern may 
relate to SCI participation, l^e more a student feels like she 
ma/ have trouble with a particular class, the more she may feel 
the need for an extra program like SCI. Thus, there should be a 
neqative relationship between expected qrade and SCI participa- 
tion. 

As stated earlier in this paper, the qoals of SCI are to 
assist the student in developing study skills for a particular 
course as well as to increase the likel ihood of success in 
college qenerally. To the extent that it achieves these goals, 
we would expect SCI Participation to be positively related to 
course grade, semester GPA, and to subsequent reenrol lment . 

Obviously, the oerformance of a student in an individual 
class will impact hi', semester GPA. This is due to the simple 
arithmetic of the GPA calculation. In addition , success in 
individual classes should result in posit ive f eelinqs reqarding 
^^irf&^\**o colleqe that make retention more likely. 

The hypothetical relationships described above may not be 
held by all developmental educators. However, we see the model 
as a plausible image of what factors may affect collecje 
performance. This model alleys a clear test oi the impact of SCI 
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Dart icipation while controlling fc-r the confoundinq effects that 
are alwavs present in evaluating student performance. 



The Data 



Th€ data includes the following variables: 

(1) years since high school. 

(2) high school grade point average (self reported). 

(3) marital status. 

(4) current semester load. 

(5) hours worked. 

(6) expected course grade (at the beginning of the 
semester ) . 

(7) number of SCI sessions attended (self reDorted). 

(8) course grade. 

(9) semester grade point average. 

(10) risk factor (high, moderate, or low, deDendinq on | 
ACT score) . 

(11) High School class rank. 

(12) reenrollment data for the following two semesters.. 

Two questionnaires were distributed during the Fall semester 
of 1985. The first questionnaire, given during the first week of 
classes, requested variables (1) through (6), as well as name, 
identification number and times the student would prefer for the 
sess ions to be held . The second questionnaire, distributed 
during the last week of classes, requested information (7) as 
well as the degree of helpfulness of the sessions, their 
expected grade, comments, and if the student did not attend 
review sessions, why. Data (8) throuqh (12) came from the 
registrar's records. 

The courses associated with SCI during the Fall of 1985 wore 
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Introduction to Sociology, Introduction to Psychology, Math for 
General Education, General Zoology, and Introduction to 
Chemistry. hese courses are freshman level courses that meet 
general education requirements for graduation. The General 
Zoology and Introduction to Chemistry courses have a large 
percent of nursing maiors. 

The data used in this analysis do not include the students 
in General Zoologv. This course was not included for several 
reasons. Pirst, the SCI program depends on the SCI leader having 
had the professor before and the Zoology course was taught by a 
first -time instructor. Second , even though there was a large 
percentage of nursing majors in the class, they were unable 
(ostensibly due to schedule conflicts) to participate in the SCI 
sessions. Third , there are certain statistical problems 
(principally mult i colli near ity) with the data from General 
Zoology. These problems make interpretation of the data from 
that course nearly impossible. 

The number of students enrolled in the four courses used in 
this evaluation totalle? 461. Most students (93%) were single. 
Of the total Pooulatiuu, 37% were employed, working an average 
of 14 hours per week. The majority of students, having graduated 
from high school the previous vear , were freshmen car rvirg an 
average of 14 semester hours. 

Most students reoorted their mean high school grade to be in 
the B range. High school rank .averaged at the 60th percentile 
(with a standard deviation of 24) and the mean ACT score was 
17.3 (with a standard deviation of 5.2). 
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Approximately 56% of the students who completed the course 
attended at least one SCI session. Those students who attended 
SCI averaged close to six sessions each durinq the semester. The 
mean course qrade was 7.381 on a four point scale and the mean 
semester GPA was 2.307 on a four point scale. 

Results 

The data described above were submitted to structural 
equation analysis to determine the effects of each variable. 
Because it is necessary to have complete data for each case, a 
listwise deletion limited the analysis to the 253 cases with 
comolete data. (Students who added late or dropped the course 
are not included.) The significant relationships are diagrammed 
in Figure 3 and Figure 4. While there are many interesting 
things that emerge from an examination of these figures 
(particularly relating to the ooints of divergence from the 
hypothetical model), the focus of the current oaper is on the 

SCI proqram. The results will be presented in two sections; f&5S8tf>*£&* 'Xt&W>i<fctf$jfc 

factors that affect SCI participation and the effects of SC I 
participation. First, however, an introduction to the 
statistical technique is required. 

An Introduction to P a th Analysis . In structural equation (or 
Dath) analysis, sta>dardized regression coefficients, or Betas 
(represented by 8), show what the impact of one variable on 
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another would be if both variables were stated in the same 
units. Technically, the path coefficients can be read as a 
percentage of a standard deviation change in the dependent 
variable that is attributable to the inderendent variable. The 
larger the magnitude of the 6, the stronger the effect. 
Becar.se the variables are all in standard units, their relative 
effects can be determined. By examining a set of relationships 
(this is where the notion of model comes in), both direct and 
indirect effects of independent variables can be examined. 

Factors Affecting SCI Participation . Overall, the hypotheti- 
cal model outlined earlier explained a total of 12.5% of the 
total variance of SCI participation. While this suggests th it 
there are probably other factors related to SCI participation, 
there are significant effects (both direct and indirect) of thf 
exogenous and endogenous var iables as presented in Table 1 . 



Table 1 

Direct, Indirect, and Total Effects 
of Factors Affecting SCI Participation 



DIRECT 
EFFECT 



INDIRECT 
EFFECT 



TOTAL 
EFFECTS 



EXOGENOUS VARIABLES 
RISK 

HIGH SCHOOL RANK 
MARITAL STATUS 
HIGH SCHOOL GRADES 
YEARS SINCE HS 



0.280 



0.170 
0.120 



-0.022 
0.002 
0. 025 



-0.026 



0.258 
O.OO'J 
0.025 
0. 170 
-0.146 



ENDOGENOUS VARIABLES 



SEMESTER LOAD 
HOURS WORKED 
EXPECTED GRADE 



0.012 



0.012 
-0.120 
0.120 



0.120 
0.120 
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Three of the exoqenous * variables have a direct, 
stati st i call y siqni Meant, impact on SCI participation , Tho 
strongest relationship exists between the level of "risk" of thn 
student and SCI (B*0.28). The more the stident is under- 
DreDared , at least usinq thi s particular measure, the more 
1 kelv he is to use SCI. On the other hand, there is also a 
direct positive effect between reported high school qrades and 
SCI participation (6-0.17). "Better" high school students are 
attracted to SCI as well. The third exoqenous variable that is 
related to SCI part i cioation is years since high school. The 
lonqer the student has been out of high school, the less likely 
he is to use SCI (fi=-0.12). Indirect effects on SCI can be 
seen coming from risk (through semester load and expected 
qrade), high school rank (through semester load), marital status 
(through hours worked and semester load), and years since hiqh 
school (throuqh hours worked). 

Two endoqenous variables also have direct impacts on SCI 
participation. The mor'i n student works, the less likely he is 
to attend SCI sessions (6=-0.12). The other imoortant 
variable is expected qrade. Students who expect to do well in 
the course are siqnificantly more likely (6=0. 12 ) to attend 
SCI sessions. There is also an indirect effect of semester load 
( throuqh hours worked ) . 

Back q round character i sties seem to have the larqest 
rela t ive impacts on SCI participation . While effects of 
endoqenous variables can be observed, these can be seen to 
"cancel each other out". 
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Table 2 

Direct, Indirect, and Total Effects 
of SCI Participation on Course Grade, 
Semester GPA f and Reenrollment 



DI RECT IND IRECT TOTAL 

EPFECT EPFECT EFFECTS 

DEPENDENT VARIABLES 

COURSE GRADE 0,140 0.140 

SEMESTER GPA 0.100 0.099 0.199 

REENROLLMENT 0.140 0.04 3 0.183 



The Impacts of SCI . As sho*n in Table 2, participating in 
SCI had significant direct effects on course grade <6=.14, 
eguation R 2 «.426J, semester GPA (6=.10, eguation 

R 2 «.745) f and reenrollment (6-. 14, equation R«=.182). It 
is important to note that these three effects are all 
independent of the other factors in the model. In other words, 
these represent the pure impacts of participation in the SCI 
program. 

In addition, SCI participation is indirectly related to 
semester GPA and reenrollment due to its relationship to course 
grade. This indirect effect on GPA is almost as large as the 
direct effect. 
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Implications for SCI 

Seven main implications can be drawn from our study. Each of 
these will be considered with more general conclusions 
following . 

1. We found that SCI serves a broad population which 
confirms our original assumptions about the type of 
student choosing to attend SCI. It is not seen as just 
for the remedial or poor student but it is seen by the 
general copulation as a means of supplementing course 
instruction . 

2. Time constraints make it difficult for married 
students to attend SCI study sessions. While married 
students may comprise a small percent of the population 
studied, scheduling evening study sessions precludes 
many from attending. Time management is a greater 
problem for the older and/or married student and this 
consideration must be dealt with in schedulinq studv 
sessions. 

3. Regardless of background, the better a student . 
expects to do in a course the more likely the student 

is to attend SCI. Students may view SCI as a means to 
achieving the better grade that they expect. SCI may no 
longer be considered as an extra but instead as an 
essential part of the course. Our challenge is to 
increase the number of students that view SCI in this 
manner as well as to deliver. 
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4. As we had hoped, there is a significant, positive 
imoact of SCI on course grade. No matter which risk 
population, high or low, the student comes from, those 
which consistently use SCI do see a positive differ- 
ence in course grade. This means that we can be more 
aggressive in marketing SCI to the students and 
faculty at Olivet. 

5. The direct effect of SCI on GPA may suggest the 
transfer of skills learned in the SCI study sessions 
to other courses, this is somewhat surprising consider- 
ing that it was suspe"te<r that study skills were not 
being taught &il that well. What would napoen if we 
did a better job of teaching study skills in SCI? 

6. The direct effect of SCI on reenrollment could be 
due to a sense of involvement resulting from the small 
grouo setting in most of the study sessions. As 
William Turnbull orooosed in the November, 1986 issue 
of the Journal of Developmental Education , involvement 
is the key to retention. As Martin et. al. state in 
thei r evaluation of supp! emental instruction. the 
higher evaluations often gi ven to professors in 
courses with SCI may be due to students attributing 
the benefits of SCI to the professor. This suggests 
that the image of the professor as caring for the 
student is translated into a form of interpersonal 
involvement . 
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Conclusions 

One reason we did this Droiect was to test the feasibility 
of usinq structural equation modeling as a Droqram evaluation 
tool. After examininq the SCI Droqram at Olivet, we feel that 
this technique has some real merit. While there may be some who 
disaqree with the measures used r with the specification of -he 
model, or with tt ■» presentation of the results (in that we 
focused only o.i SCI), we feel that the benefits of this approach 
outweiqh any complaints against it. 

The findinq that SCI had a direct benefit on student 
outcomes was important for two reasons. First, it can be 
influential in decidinq "political- issues inside the 
university. Because we can demonstrate a real impact of the 
oroqram, we can protect its future development in terms of 
continuation and potential expansion. The other reason this was 
an important findinq is that the ONU model for Supplemental 
Course Instruction is a variant of the UMKC model. Because *ve 
use underqraduates rather than qraduate students and cannot 

» 

orovide intensive supervision, we were worried about qeneraliz- 
inj from the results shown in Blanc et. al_.. However, since we 
can show direct benefits, even qiven these proqram variations, 
we feel confident that our model can continue to be effective in 
assi stinq students of all ability levels at beinq more 
successful in their colleqe experience. 
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